Skip to content

feat(run-engine,run-store): completed-waitpoint envelope, read-time resolver, and the fail-loud coverage check - #4779

Open
d-cs wants to merge 23 commits into
mainfrom
feat/waitpoint-envelope-resolver-tri-13441
Open

feat(run-engine,run-store): completed-waitpoint envelope, read-time resolver, and the fail-loud coverage check#4779
d-cs wants to merge 23 commits into
mainfrom
feat/waitpoint-envelope-resolver-tri-13441

Conversation

@d-cs

@d-cs d-cs commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Defines the completed-waitpoint record shape, writes it at the resume appends, and rebuilds CompletedWaitpoint[] from it at read time.

The record set lives in an immutable per-cycle Redis key: an ordered waitpoint id list plus one envelope per distinct id. A waitpoint's output is never copied through the snapshot chain — a record carries an inline value, a reference to an already-offloaded one, or a marker saying the value is re-readable from the completing run.

What is here

  • The envelope, byte-compatible with enhanceExecutionSnapshotWithWaitpoints, which is what the executor consumes today. A differential suite compares the two over real rows rather than against a literal, so a drift in either side shows up as a diff.
  • The resume-time write, at both appending branches of continueRunIfUnblocked.
  • The read-time resolver, which iterates the records rather than the order — the order holds only batch-indexed ids, so iterating it would silently drop every index-less wait.
  • Mixed-snapshot support: a snapshot holding legacy and store-format ids resolves both halves, each reading its index from the same order.
  • Fail-loud: an id that no half accounts for, or that both claim, refuses rather than resuming short.

Merge test

Merged alone, nothing is observable.

The record build sits behind an O(1) predicate that defaults to never, so no resume reaches it. Nothing consumes a record set either: the resolver has no production caller and is not exported from the package index. The wiring is a later ticket.

The append script is modified, and organisations already on the snapshot store execute it. Those changes are behaviour-preserving for the paths they take: the new and healthy carry-forward branches are byte-identical to main, and the new branch needs a failed head probe to reach.

Performance

No added I/O on any production path at any dial setting. A record set is read only where one can exist, and never on a copy-forward append — the append script sources it from the cycle it replaces, in the one branch that needs it, atomically with the mint.

Testing

Roughly 190 automated tests. The ones worth knowing about:

  • a differential suite against the existing hydration, over real waitpoint rows;
  • a round-trip through a real cycle key: envelopes from Postgres rows, written through the real store, id lists read back through the store's read API, then resolved and compared with the oracle;
  • an exhaustive walk of the reachable output state space, which is where every defect on this branch turned out to live.

Mutation-verified at a dozen points, including that emptying the envelope build fails four of the five round-trip cases.

Handoff to the wiring ticket

Two things this PR cannot supply and the next one must:

  1. A Postgres fallback when the resolver throws. findLatestExecutionSnapshot falls back on a miss and on a dangling cycle, but a throw propagates. Unhandled, one unresolvable cycle makes a run's execution data unreadable rather than merely unresumable, while Postgres still holds the join rows.
  2. The records-enabled predicate, and the store coordinator arm. Store-format minting should follow the snapshot store rather than lead it: the engine gate keys on id format, so the first store-format mint is what starts the cost, not the first store flip.

@changeset-bot

changeset-bot Bot commented Aug 25, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: dbefc5e

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

The run engine now reads completed waitpoint envelopes from Postgres or Redis and converts them into deduplicated completed-waitpoint records. Resolver logic reconstructs executor-compatible waitpoints, including outputs, metadata, indexes, and validation errors. Resume and enqueue paths forward these records into execution snapshots. The run store accepts the expanded snapshot inputs, and the Redis snapshot decorator persists records across staged, carried-forward, and replacement cycles. Tests cover equivalence, output handling, validation, Redis reads, and snapshot persistence.

Merge Risk: 🟡 Moderate · up to 5b4b0

This PR adds completed-waitpoint persistence and read-time reconstruction, but the current head is not merge-ready because it depends on #4765 and still has open concerns that could cause incorrect resume data, stale idempotency metadata, or unnecessary runtime reads; required repository markers also need to be added.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 37.50% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 88 functions across 52 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the main changes: completed-waitpoint envelope support, read-time resolution, and fail-loud coverage.
Description check ✅ Passed The description gives detailed scope, implementation behavior, testing, performance impact, and follow-up requirements. It does not use the repository template or include the required issue reference,…
Full details: Description check

Explanation

The description gives detailed scope, implementation behavior, testing, performance impact, and follow-up requirements. It does not use the repository template or include the required issue reference, checklist, changelog, or screenshots sections, but the core information is complete and relevant.

✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/waitpoint-envelope-resolver-tri-13441

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@d-cs d-cs self-assigned this Aug 25, 2026
Base automatically changed from feat/snapshot-store-decorator-tri-13449 to main August 26, 2026 13:20
@d-cs
d-cs marked this pull request as ready for review August 26, 2026 14:30
@d-cs
d-cs force-pushed the feat/waitpoint-envelope-resolver-tri-13441 branch from 7154c3c to 7f9da73 Compare August 26, 2026 14:36
@pkg-pr-new

pkg-pr-new Bot commented Aug 26, 2026

Copy link
Copy Markdown

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@9e8dc9d

trigger.dev

npm i https://pkg.pr.new/trigger.dev@9e8dc9d

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@9e8dc9d

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@9e8dc9d

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@9e8dc9d

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@9e8dc9d

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@9e8dc9d

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@9e8dc9d

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@9e8dc9d

commit: 9e8dc9d

devin-ai-integration[bot]

This comment was marked as resolved.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

🧹 Nitpick comments (5)
internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts (3)

156-157: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

This assertion cannot fail; express it against the crash count.

Line 146 already asserts pgRun.attemptNumber is exactly 1, so 1 - 1 <= 1 always holds. The comment says the line proves the per-crash bound, but it proves nothing. The last test in this file states the same property correctly against faults.fired(...).

♻️ Proposed change
-        // The bound: one crash costs at most one attempt number.
-        expect(pgRun.attemptNumber! - 1).toBeLessThanOrEqual(1);
+        // The bound: pgAttempt - maxLoggedAttempt <= crashCount.
+        expect((pgRun.attemptNumber ?? 0) - 1).toBeLessThanOrEqual(
+          faults.fired("afterPgBeforeRedis")
+        );

383-384: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

The test name claims two crashes, but the assertion accepts one.

expect(crashes).toBeGreaterThanOrEqual(1) passes when only one fault fires. The two-crash scenario named in the title and in the comment on Line 390 is then never exercised, and the bound assertion on Line 391 degrades to the single-crash case. This is the same silent-pass failure mode the fired() guards elsewhere in this file exist to prevent.

Assert the exact expected count.

♻️ Proposed change
         const crashes = faults.fired("afterPgBeforeRedis");
-        expect(crashes).toBeGreaterThanOrEqual(1);
+        expect(crashes).toBe(2);

197-204: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Three dequeue sites index dequeued[0]! without the length assertion this file uses elsewhere. The first and third tests assert expect(dequeued.length).toBe(1) before indexing. The other three do not, so an empty queue produces a TypeError instead of the intended assertion failure.

  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts#L197-L204: add expect(dequeued.length).toBe(1) after the dequeue call in the afterRedisBirthBeforePg test.
  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts#L315-L320: add the same assertion after the dequeue call in the stale-snapshot test.
  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts#L364-L371: add the same assertion after the dequeue call in the two-crash test.
internal-packages/run-store/src/taskRunExecutionSnapshotStore.births.test.ts (1)

232-266: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

This test cannot fail if the birth omits the completion TTL.

The comment states the invariant: a born-terminal run never transitions again, so the birth itself must apply the completion TTL. The assertions only check that both keyspaces are readable. They pass whether or not any expiry was set, and the non-terminal run exists only as an unused comparison.

Assert the expiry directly with a raw Redis client, as taskRunExecutionSnapshotStore.waitpointCycles.test.ts does for cycle keys: expect a positive pttl on the terminal run's key and -1 on the non-terminal run's key.

internal-packages/run-store/src/redisSnapshotStore.ts (1)

459-494: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Extract the shared row-decode and head-resolution block.

getSince and getSinceCreatedAt now carry the same loop, the same headSurvived tracking, the same rows.reverse(), and the same head attribution. The offsets (i = 3, stride 4) and the reply layout must stay identical in both, so a future change to the Lua reply shape must be applied twice. Extract one private helper that takes the reply and returns { entries, headWaitpointIds }.

♻️ Sketch
`#decodeSinceReply`(
  reply: string[],
  environmentId: string | undefined,
  runId: string
): { entries: SnapshotRead[]; headWaitpointIds: WaitpointIds } {
  const headOrder = reply[1] ?? "";
  const headDistinct = reply[2] ?? "";
  const rows: SnapshotRead[] = [];
  let headSurvived = false;
  for (let i = 3; i + 3 < reply.length; i += 4) {
    const decoded = this.#decode(
      [reply[i], reply[i + 1], reply[i + 2], reply[i + 3], ""],
      environmentId,
      runId,
      false
    );
    if (decoded) {
      rows.push(decoded);
      if (i === 3) headSurvived = true;
    }
  }
  rows.reverse();
  const head = headSurvived ? rows[rows.length - 1] : undefined;
  const headWaitpointIds = decodeWaitpointIds(
    head !== undefined,
    head ? headOrder : "",
    head ? headDistinct : ""
  );
  if (head) {
    head.completedWaitpointIds = headWaitpointIds;
    if (head.cycle) {
      this.#checkCycleMismatch(runId, head.cycle.count, headWaitpointIds.order.length);
    }
  }
  return { entries: rows, headWaitpointIds };
}

Also applies to: 503-557


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 2802deac-da94-48dd-8065-ab7bc7c0da5c

📥 Commits

Reviewing files that changed from the base of the PR and between 02e6157 and 7154c3c.

📒 Files selected for processing (46)
  • internal-packages/run-engine/src/engine/systems/enqueueSystem.ts
  • internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts
  • internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
  • internal-packages/run-engine/src/engine/tests/finalRunStatusParity.test.ts
  • internal-packages/run-engine/src/engine/tests/helpers/decoratedStore.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreReadGate.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointEquivalence.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/legacyPostgresCoordinator.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/types.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotId.test.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotTimestamps.test.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotWrites.test.ts
  • internal-packages/run-store/src/PostgresRunStore.ts
  • internal-packages/run-store/src/delegatingRunStore.forwarding.test.ts
  • internal-packages/run-store/src/delegatingRunStore.test.ts
  • internal-packages/run-store/src/delegatingRunStore.ts
  • internal-packages/run-store/src/index.ts
  • internal-packages/run-store/src/redisSnapshotStore.sinceCreatedAt.test.ts
  • internal-packages/run-store/src/redisSnapshotStore.ts
  • internal-packages/run-store/src/runStoreMethodNames.ts
  • internal-packages/run-store/src/snapshotEntry.parity.test.ts
  • internal-packages/run-store/src/snapshotEntry.test.ts
  • internal-packages/run-store/src/snapshotEntry.ts
  • internal-packages/run-store/src/snapshotFaultInjection.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.test.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.ts
  • internal-packages/run-store/src/snapshotReadShapes.test.ts
  • internal-packages/run-store/src/snapshotReadShapes.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.births.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.off.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.readCohort.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.reads.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.staging.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.transitions.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointCycles.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointRecords.test.ts
  • internal-packages/run-store/src/testFixtures/snapshotIdFixture.ts
  • internal-packages/run-store/src/types.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 7 remain after this review.

📜 Review details
🧰 Additional context used
📓 Path-based instructions (9)
We use vitest exclusively. **Never mock anything** - use testcontainers instead.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • internal-packages/run-engine/src/engine/tests/finalRunStatusParity.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.test.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.readCohort.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.births.test.ts
  • internal-packages/run-store/src/delegatingRunStore.test.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotId.test.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotTimestamps.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts
  • internal-packages/run-store/src/delegatingRunStore.forwarding.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.reads.test.ts
  • internal-packages/run-store/src/redisSnapshotStore.sinceCreatedAt.test.ts
  • internal-packages/run-store/src/snapshotReadShapes.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointEquivalence.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.staging.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointRecords.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointCycles.test.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotWrites.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.transitions.test.ts
  • internal-packages/run-store/src/snapshotEntry.parity.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreReadGate.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.off.test.ts
  • internal-packages/run-store/src/snapshotEntry.test.ts
**Prefer static imports over dynamic imports.** Only use dynamic `import()` when:

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • internal-packages/run-engine/src/engine/tests/finalRunStatusParity.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.test.ts
  • internal-packages/run-store/src/index.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.readCohort.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.births.test.ts
  • internal-packages/run-store/src/delegatingRunStore.test.ts
  • internal-packages/run-store/src/runStoreMethodNames.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotId.test.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotTimestamps.test.ts
  • internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts
  • internal-packages/run-store/src/snapshotFaultInjection.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts
  • internal-packages/run-store/src/delegatingRunStore.forwarding.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.reads.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/legacyPostgresCoordinator.ts
  • internal-packages/run-store/src/redisSnapshotStore.sinceCreatedAt.test.ts
  • internal-packages/run-store/src/snapshotReadShapes.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointEquivalence.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.staging.test.ts
  • internal-packages/run-store/src/testFixtures/snapshotIdFixture.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.ts
  • internal-packages/run-engine/src/engine/tests/helpers/decoratedStore.ts
  • internal-packages/run-store/src/snapshotEntry.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointRecords.test.ts
  • internal-packages/run-engine/src/engine/systems/enqueueSystem.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/types.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointCycles.test.ts
  • internal-packages/run-store/src/snapshotReadShapes.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotWrites.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.transitions.test.ts
  • internal-packages/run-store/src/types.ts
  • internal-packages/run-store/src/snapshotEntry.parity.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreReadGate.test.ts
  • internal-packages/run-store/src/PostgresRunStore.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.off.test.ts
  • internal-packages/run-store/src/delegatingRunStore.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.ts
  • internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
  • internal-packages/run-store/src/redisSnapshotStore.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.ts
  • internal-packages/run-store/src/snapshotEntry.test.ts
Add crumbs as you write code — not just when debugging. Mark lines with

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • internal-packages/run-engine/src/engine/tests/finalRunStatusParity.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.test.ts
  • internal-packages/run-store/src/index.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.readCohort.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.births.test.ts
  • internal-packages/run-store/src/delegatingRunStore.test.ts
  • internal-packages/run-store/src/runStoreMethodNames.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotId.test.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotTimestamps.test.ts
  • internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts
  • internal-packages/run-store/src/snapshotFaultInjection.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts
  • internal-packages/run-store/src/delegatingRunStore.forwarding.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.reads.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/legacyPostgresCoordinator.ts
  • internal-packages/run-store/src/redisSnapshotStore.sinceCreatedAt.test.ts
  • internal-packages/run-store/src/snapshotReadShapes.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointEquivalence.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.staging.test.ts
  • internal-packages/run-store/src/testFixtures/snapshotIdFixture.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.ts
  • internal-packages/run-engine/src/engine/tests/helpers/decoratedStore.ts
  • internal-packages/run-store/src/snapshotEntry.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointRecords.test.ts
  • internal-packages/run-engine/src/engine/systems/enqueueSystem.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/types.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointCycles.test.ts
  • internal-packages/run-store/src/snapshotReadShapes.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotWrites.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.transitions.test.ts
  • internal-packages/run-store/src/types.ts
  • internal-packages/run-store/src/snapshotEntry.parity.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreReadGate.test.ts
  • internal-packages/run-store/src/PostgresRunStore.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.off.test.ts
  • internal-packages/run-store/src/delegatingRunStore.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.ts
  • internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
  • internal-packages/run-store/src/redisSnapshotStore.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.ts
  • internal-packages/run-store/src/snapshotEntry.test.ts
Implement tests for RunEngine in `src/engine/tests/` using testcontainers for Redis and PostgreSQL containerization

📄 CodeRabbit inference engine (internal-packages/run-engine/CLAUDE.md)

Files:

  • internal-packages/run-engine/src/engine/tests/finalRunStatusParity.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreReadGate.test.ts
Integrate OpenTelemetry tracer and meter instrumentation in RunEngine systems for observability

📄 CodeRabbit inference engine (internal-packages/run-engine/CLAUDE.md)

Files:

  • internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts
  • internal-packages/run-engine/src/engine/systems/enqueueSystem.ts
  • internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
Use vitest for all tests in the Trigger.dev repository

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • internal-packages/run-engine/src/engine/tests/finalRunStatusParity.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.test.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.readCohort.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.births.test.ts
  • internal-packages/run-store/src/delegatingRunStore.test.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotId.test.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotTimestamps.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts
  • internal-packages/run-store/src/delegatingRunStore.forwarding.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.reads.test.ts
  • internal-packages/run-store/src/redisSnapshotStore.sinceCreatedAt.test.ts
  • internal-packages/run-store/src/snapshotReadShapes.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointEquivalence.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.staging.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointRecords.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointCycles.test.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotWrites.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.transitions.test.ts
  • internal-packages/run-store/src/snapshotEntry.parity.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreReadGate.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.off.test.ts
  • internal-packages/run-store/src/snapshotEntry.test.ts
Use function declarations instead of default exports

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • internal-packages/run-engine/src/engine/tests/finalRunStatusParity.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.test.ts
  • internal-packages/run-store/src/index.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.readCohort.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.births.test.ts
  • internal-packages/run-store/src/delegatingRunStore.test.ts
  • internal-packages/run-store/src/runStoreMethodNames.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotId.test.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotTimestamps.test.ts
  • internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts
  • internal-packages/run-store/src/snapshotFaultInjection.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts
  • internal-packages/run-store/src/delegatingRunStore.forwarding.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.reads.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/legacyPostgresCoordinator.ts
  • internal-packages/run-store/src/redisSnapshotStore.sinceCreatedAt.test.ts
  • internal-packages/run-store/src/snapshotReadShapes.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointEquivalence.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.staging.test.ts
  • internal-packages/run-store/src/testFixtures/snapshotIdFixture.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.ts
  • internal-packages/run-engine/src/engine/tests/helpers/decoratedStore.ts
  • internal-packages/run-store/src/snapshotEntry.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointRecords.test.ts
  • internal-packages/run-engine/src/engine/systems/enqueueSystem.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/types.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointCycles.test.ts
  • internal-packages/run-store/src/snapshotReadShapes.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotWrites.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.transitions.test.ts
  • internal-packages/run-store/src/types.ts
  • internal-packages/run-store/src/snapshotEntry.parity.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreReadGate.test.ts
  • internal-packages/run-store/src/PostgresRunStore.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.off.test.ts
  • internal-packages/run-store/src/delegatingRunStore.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.ts
  • internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
  • internal-packages/run-store/src/redisSnapshotStore.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.ts
  • internal-packages/run-store/src/snapshotEntry.test.ts
Use types over interfaces for TypeScript

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Files:

  • internal-packages/run-engine/src/engine/tests/finalRunStatusParity.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.test.ts
  • internal-packages/run-store/src/index.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.readCohort.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.births.test.ts
  • internal-packages/run-store/src/delegatingRunStore.test.ts
  • internal-packages/run-store/src/runStoreMethodNames.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotId.test.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotTimestamps.test.ts
  • internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts
  • internal-packages/run-store/src/snapshotFaultInjection.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts
  • internal-packages/run-store/src/delegatingRunStore.forwarding.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.reads.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/legacyPostgresCoordinator.ts
  • internal-packages/run-store/src/redisSnapshotStore.sinceCreatedAt.test.ts
  • internal-packages/run-store/src/snapshotReadShapes.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointEquivalence.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.staging.test.ts
  • internal-packages/run-store/src/testFixtures/snapshotIdFixture.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.ts
  • internal-packages/run-engine/src/engine/tests/helpers/decoratedStore.ts
  • internal-packages/run-store/src/snapshotEntry.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointRecords.test.ts
  • internal-packages/run-engine/src/engine/systems/enqueueSystem.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/types.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointCycles.test.ts
  • internal-packages/run-store/src/snapshotReadShapes.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotWrites.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.transitions.test.ts
  • internal-packages/run-store/src/types.ts
  • internal-packages/run-store/src/snapshotEntry.parity.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreReadGate.test.ts
  • internal-packages/run-store/src/PostgresRunStore.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.off.test.ts
  • internal-packages/run-store/src/delegatingRunStore.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.ts
  • internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
  • internal-packages/run-store/src/redisSnapshotStore.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.ts
  • internal-packages/run-store/src/snapshotEntry.test.ts
When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs

📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)

Files:

  • internal-packages/run-engine/src/engine/tests/finalRunStatusParity.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.test.ts
  • internal-packages/run-store/src/index.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.readCohort.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.births.test.ts
  • internal-packages/run-store/src/delegatingRunStore.test.ts
  • internal-packages/run-store/src/runStoreMethodNames.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotId.test.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotTimestamps.test.ts
  • internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts
  • internal-packages/run-store/src/snapshotFaultInjection.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts
  • internal-packages/run-store/src/delegatingRunStore.forwarding.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.reads.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/legacyPostgresCoordinator.ts
  • internal-packages/run-store/src/redisSnapshotStore.sinceCreatedAt.test.ts
  • internal-packages/run-store/src/snapshotReadShapes.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointEquivalence.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.staging.test.ts
  • internal-packages/run-store/src/testFixtures/snapshotIdFixture.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.ts
  • internal-packages/run-engine/src/engine/tests/helpers/decoratedStore.ts
  • internal-packages/run-store/src/snapshotEntry.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointRecords.test.ts
  • internal-packages/run-engine/src/engine/systems/enqueueSystem.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/types.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointCycles.test.ts
  • internal-packages/run-store/src/snapshotReadShapes.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotWrites.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.transitions.test.ts
  • internal-packages/run-store/src/types.ts
  • internal-packages/run-store/src/snapshotEntry.parity.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreReadGate.test.ts
  • internal-packages/run-store/src/PostgresRunStore.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.off.test.ts
  • internal-packages/run-store/src/delegatingRunStore.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.ts
  • internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
  • internal-packages/run-store/src/redisSnapshotStore.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.ts
  • internal-packages/run-store/src/snapshotEntry.test.ts
🧠 Learnings (5)
📚 Learning: 2026-06-04T18:16:35.386Z
Learnt from: nicktrn
Repo: triggerdotdev/trigger.dev PR: 3836
File: apps/supervisor/src/backpressure/backpressureMonitor.ts:3-5
Timestamp: 2026-06-04T18:16:35.386Z
Learning: When reviewing TypeScript in this repo, apply the rule “prefer type aliases over interfaces” only to data/object shapes and union/intersection type modeling. If an interface is being used as a behavioral contract for collaborators to implement (e.g., method-shape interfaces that define required behavior, such as `BackpressureLogger` / `BackpressureSignalSource` in `apps/supervisor/src/backpressure/backpressureMonitor.ts`), keep it as an `interface` and do not flag it as a type-alias-vs-interface violation.

Applied to files:

  • internal-packages/run-store/src/runStoreMethodNames.ts
  • internal-packages/run-store/src/testFixtures/snapshotIdFixture.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/types.ts
  • internal-packages/run-store/src/snapshotReadShapes.ts
  • internal-packages/run-store/src/types.ts
  • internal-packages/run-store/src/delegatingRunStore.ts
📚 Learning: 2026-06-16T09:19:47.637Z
Learnt from: d-cs
Repo: triggerdotdev/trigger.dev PR: 3960
File: apps/webapp/test/prismaInfrastructureErrorCapture.test.ts:0-0
Timestamp: 2026-06-16T09:19:47.637Z
Learning: In this repo’s Vitest setup, `vitest.config.ts` uses `globals: true`, so identifiers like `vi`, `describe`, `it`, and `expect` are available as globals in Vitest test files. During code review, do not flag missing `vi`/`describe`/`it`/`expect` imports as a runtime error or correctness issue when they’re used in `*.test.ts/tsx` or `*.spec.ts/tsx` files. Explicit imports are still preferred for consistency, but they’re not required for runtime behavior.

Applied to files:

  • internal-packages/run-store/src/PostgresRunStore.snapshotTimestamps.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreReadGate.test.ts
📚 Learning: 2026-08-15T17:58:37.120Z
Learnt from: 1stvamp
Repo: triggerdotdev/trigger.dev PR: 4628
File: internal-packages/run-engine/src/run-queue/tests/ckWildcardKey.test.ts:0-0
Timestamp: 2026-08-15T17:58:37.120Z
Learning: In internal-packages/run-engine test files, use Vitest's established global test API when Vitest globals are enabled. Do not import describe from node:test, because it shadows Vitest's global describe and registers test blocks with Node's test runner; remove the node:test import rather than replacing it with a Vitest import.

Applied to files:

  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreReadGate.test.ts
📚 Learning: 2026-06-23T13:04:21.413Z
Learnt from: carderne
Repo: triggerdotdev/trigger.dev PR: 4023
File: apps/webapp/app/services/upsertBranch.server.ts:14-18
Timestamp: 2026-06-23T13:04:21.413Z
Learning: In TypeScript, it’s valid to `import { type X }` and then use `typeof X` in a type-only position, e.g. `type Alias = z.infer<typeof X>`. The `type` modifier suppresses the runtime import, but the type checker still has the full exported type so `z.infer<typeof X>` can resolve correctly. In code reviews, don’t flag this as a TypeScript compile error as long as `typeof X` is used in a type context (e.g., with `z.infer`, `type` aliases, generics), not as a runtime value.

Applied to files:

  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointEquivalence.test.ts
📚 Learning: 2026-05-18T14:40:02.173Z
Learnt from: ericallam
Repo: triggerdotdev/trigger.dev PR: 3658
File: packages/core/src/v3/realtimeStreams/manager.test.ts:1-147
Timestamp: 2026-05-18T14:40:02.173Z
Learning: In the triggerdotdev/trigger.dev repo, the policy “Never mock anything — use testcontainers instead” should only be enforced for integration tests that interact with real external services (e.g., Redis, Postgres) via actual infrastructure. For unit tests that exercise pure in-memory logic (e.g., cache semantics) it is OK to stub collaborators such as `ApiClient` using Vitest (`vi.fn()`) to assert call counts or control behavior. Do not flag `vi.fn()`-based `ApiClient` stubs in unit tests as violations of the testcontainers policy.

Applied to files:

  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.staging.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointRecords.test.ts
🪛 ast-grep (0.45.2)
internal-packages/run-store/src/delegatingRunStore.forwarding.test.ts

[warning] 168-168: Regular expression constructed from variable input detected. This can lead to Regular Expression Denial of Service (ReDoS) attacks if the variable contains malicious patterns. Use libraries like 'recheck' to validate regex safety or use static patterns.
Context: new RegExp(return this\\.delegate\\.${name}\\(([^;]*)\\);)
Note: [CWE-1333] Inefficient Regular Expression Complexity

(regexp-from-variable)

🪛 OpenGrep (1.26.0)
internal-packages/run-store/src/delegatingRunStore.forwarding.test.ts

[ERROR] 117-117: Dynamic command passed to child_process.exec/execSync. Use child_process.execFile or spawn with an argument array instead.

(coderabbit.command-injection.exec-js)


[ERROR] 139-139: Dynamic command passed to child_process.exec/execSync. Use child_process.execFile or spawn with an argument array instead.

(coderabbit.command-injection.exec-js)


[ERROR] 169-169: Dynamic command passed to child_process.exec/execSync. Use child_process.execFile or spawn with an argument array instead.

(coderabbit.command-injection.exec-js)

🔇 Additional comments (41)
internal-packages/run-store/src/snapshotOrphanSweeper.ts (8)

25-58: LGTM!


92-111: LGTM!


116-144: LGTM!


184-207: LGTM!


210-229: LGTM!


247-266: LGTM!


272-338: LGTM!


149-163: 🗄️ Data Integrity & Integration

No change needed. RunStore.findRunsByIds returns an ID-keyed Map, and its implementation keys entries by the internal id. rows.get(runId) uses the correct key.

internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts (3)

32-58: LGTM!


60-104: LGTM!


226-279: LGTM!

internal-packages/run-store/src/delegatingRunStore.ts (1)

51-750: LGTM!

internal-packages/run-store/src/delegatingRunStore.test.ts (1)

76-88: 🎯 Functional Correctness

No change needed. runStoreMethodNames.ts checks both directions against keyof RunStore, so missing or extra member names cause a TypeScript error.

internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts (1)

13-13: LGTM!

Also applies to: 452-452, 474-474, 497-497

internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.ts (1)

12-26: LGTM!

Also applies to: 49-124, 126-149

internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.test.ts (1)

9-28: LGTM!

Also applies to: 30-80, 82-178, 180-259, 264-332

internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointEquivalence.test.ts (1)

20-123: LGTM!

Also applies to: 125-330

internal-packages/run-engine/src/engine/tests/finalRunStatusParity.test.ts (1)

8-20: LGTM!

internal-packages/run-store/src/PostgresRunStore.snapshotId.test.ts (1)

10-149: LGTM!

internal-packages/run-store/src/PostgresRunStore.snapshotWrites.test.ts (1)

16-96: LGTM!

Also applies to: 98-202, 204-235, 237-311

internal-packages/run-store/src/snapshotOrphanSweeper.test.ts (1)

21-34: LGTM!

Also applies to: 37-117, 119-194, 196-272, 274-352, 354-396, 398-429

internal-packages/run-store/src/testFixtures/snapshotIdFixture.ts (1)

8-77: LGTM!

Also applies to: 79-149, 151-167

internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointRecords.test.ts (1)

25-109: LGTM!

Also applies to: 111-143, 147-161, 165-184, 186-218

internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.test.ts (1)

1839-1975: LGTM!

internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.ts (1)

509-563: LGTM!

Also applies to: 596-629

internal-packages/run-store/src/PostgresRunStore.snapshotTimestamps.test.ts (1)

113-132: LGTM!

internal-packages/run-store/src/taskRunExecutionSnapshotStore.reads.test.ts (1)

79-420: LGTM!

internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointCycles.test.ts (1)

96-544: LGTM!

internal-packages/run-engine/src/engine/tests/helpers/decoratedStore.ts (1)

31-75: LGTM!

internal-packages/run-engine/src/engine/waitpointCoordinator/legacyPostgresCoordinator.ts (1)

104-149: 🗄️ Data Integrity & Integration

Do not add the COMPLETED predicate.

WaitpointSystem calls this method only after readRunBlockState reports all blockers as COMPLETED. Both reads pass the writer client, which routes to the owning primary, so a pending row cannot reach this mapping through the caller path.

internal-packages/run-store/src/redisSnapshotStore.ts (1)

32-44: LGTM!

Also applies to: 286-319, 426-426, 581-601, 617-623, 643-645, 680-680, 750-750, 762-762, 775-837, 867-900, 923-923, 949-957

internal-packages/run-store/src/taskRunExecutionSnapshotStore.ts (1)

118-138: LGTM!

Also applies to: 157-221, 227-257, 263-425, 447-566, 584-627, 636-651, 666-795, 804-866, 872-911, 913-962

internal-packages/run-store/src/snapshotFaultInjection.ts (1)

9-45: LGTM!

internal-packages/run-store/src/index.ts (1)

6-10: LGTM!

internal-packages/run-store/src/redisSnapshotStore.sinceCreatedAt.test.ts (1)

11-41: LGTM!

Also applies to: 43-193

internal-packages/run-store/src/snapshotReadShapes.ts (1)

13-27: LGTM!

Also applies to: 29-51, 53-95

internal-packages/run-store/src/taskRunExecutionSnapshotStore.off.test.ts (1)

10-38: LGTM!

Also applies to: 40-100

internal-packages/run-store/src/taskRunExecutionSnapshotStore.staging.test.ts (2)

20-65: LGTM!

Also applies to: 67-251


253-285: 📐 Maintainability & Code Quality

No change needed. PostgresRunStore.forWaitpointCompletion ignores the waitpoint ID and context and returns this, so an unknown waitpoint ID does not fail before the assertions.

internal-packages/run-store/src/snapshotReadShapes.test.ts (1)

7-17: LGTM!

Also applies to: 19-151

internal-packages/run-store/src/taskRunExecutionSnapshotStore.transitions.test.ts (1)

26-134: LGTM!

Also applies to: 136-454

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.

Actionable comments posted: 8

🧹 Nitpick comments (5)
internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts (3)

156-157: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

This assertion cannot fail; express it against the crash count.

Line 146 already asserts pgRun.attemptNumber is exactly 1, so 1 - 1 <= 1 always holds. The comment says the line proves the per-crash bound, but it proves nothing. The last test in this file states the same property correctly against faults.fired(...).

♻️ Proposed change
-        // The bound: one crash costs at most one attempt number.
-        expect(pgRun.attemptNumber! - 1).toBeLessThanOrEqual(1);
+        // The bound: pgAttempt - maxLoggedAttempt <= crashCount.
+        expect((pgRun.attemptNumber ?? 0) - 1).toBeLessThanOrEqual(
+          faults.fired("afterPgBeforeRedis")
+        );

383-384: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

The test name claims two crashes, but the assertion accepts one.

expect(crashes).toBeGreaterThanOrEqual(1) passes when only one fault fires. The two-crash scenario named in the title and in the comment on Line 390 is then never exercised, and the bound assertion on Line 391 degrades to the single-crash case. This is the same silent-pass failure mode the fired() guards elsewhere in this file exist to prevent.

Assert the exact expected count.

♻️ Proposed change
         const crashes = faults.fired("afterPgBeforeRedis");
-        expect(crashes).toBeGreaterThanOrEqual(1);
+        expect(crashes).toBe(2);

197-204: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Three dequeue sites index dequeued[0]! without the length assertion this file uses elsewhere. The first and third tests assert expect(dequeued.length).toBe(1) before indexing. The other three do not, so an empty queue produces a TypeError instead of the intended assertion failure.

  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts#L197-L204: add expect(dequeued.length).toBe(1) after the dequeue call in the afterRedisBirthBeforePg test.
  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts#L315-L320: add the same assertion after the dequeue call in the stale-snapshot test.
  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts#L364-L371: add the same assertion after the dequeue call in the two-crash test.
internal-packages/run-store/src/taskRunExecutionSnapshotStore.births.test.ts (1)

232-266: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

This test cannot fail if the birth omits the completion TTL.

The comment states the invariant: a born-terminal run never transitions again, so the birth itself must apply the completion TTL. The assertions only check that both keyspaces are readable. They pass whether or not any expiry was set, and the non-terminal run exists only as an unused comparison.

Assert the expiry directly with a raw Redis client, as taskRunExecutionSnapshotStore.waitpointCycles.test.ts does for cycle keys: expect a positive pttl on the terminal run's key and -1 on the non-terminal run's key.

internal-packages/run-store/src/redisSnapshotStore.ts (1)

459-494: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Extract the shared row-decode and head-resolution block.

getSince and getSinceCreatedAt now carry the same loop, the same headSurvived tracking, the same rows.reverse(), and the same head attribution. The offsets (i = 3, stride 4) and the reply layout must stay identical in both, so a future change to the Lua reply shape must be applied twice. Extract one private helper that takes the reply and returns { entries, headWaitpointIds }.

♻️ Sketch
`#decodeSinceReply`(
  reply: string[],
  environmentId: string | undefined,
  runId: string
): { entries: SnapshotRead[]; headWaitpointIds: WaitpointIds } {
  const headOrder = reply[1] ?? "";
  const headDistinct = reply[2] ?? "";
  const rows: SnapshotRead[] = [];
  let headSurvived = false;
  for (let i = 3; i + 3 < reply.length; i += 4) {
    const decoded = this.#decode(
      [reply[i], reply[i + 1], reply[i + 2], reply[i + 3], ""],
      environmentId,
      runId,
      false
    );
    if (decoded) {
      rows.push(decoded);
      if (i === 3) headSurvived = true;
    }
  }
  rows.reverse();
  const head = headSurvived ? rows[rows.length - 1] : undefined;
  const headWaitpointIds = decodeWaitpointIds(
    head !== undefined,
    head ? headOrder : "",
    head ? headDistinct : ""
  );
  if (head) {
    head.completedWaitpointIds = headWaitpointIds;
    if (head.cycle) {
      this.#checkCycleMismatch(runId, head.cycle.count, headWaitpointIds.order.length);
    }
  }
  return { entries: rows, headWaitpointIds };
}

Also applies to: 503-557


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 2802deac-da94-48dd-8065-ab7bc7c0da5c

📥 Commits

Reviewing files that changed from the base of the PR and between 02e6157 and 7154c3c.

📒 Files selected for processing (46)
  • internal-packages/run-engine/src/engine/systems/enqueueSystem.ts
  • internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts
  • internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
  • internal-packages/run-engine/src/engine/tests/finalRunStatusParity.test.ts
  • internal-packages/run-engine/src/engine/tests/helpers/decoratedStore.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts
  • internal-packages/run-engine/src/engine/tests/snapshotStoreReadGate.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointEquivalence.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/legacyPostgresCoordinator.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.test.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.ts
  • internal-packages/run-engine/src/engine/waitpointCoordinator/types.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotId.test.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotTimestamps.test.ts
  • internal-packages/run-store/src/PostgresRunStore.snapshotWrites.test.ts
  • internal-packages/run-store/src/PostgresRunStore.ts
  • internal-packages/run-store/src/delegatingRunStore.forwarding.test.ts
  • internal-packages/run-store/src/delegatingRunStore.test.ts
  • internal-packages/run-store/src/delegatingRunStore.ts
  • internal-packages/run-store/src/index.ts
  • internal-packages/run-store/src/redisSnapshotStore.sinceCreatedAt.test.ts
  • internal-packages/run-store/src/redisSnapshotStore.ts
  • internal-packages/run-store/src/runStoreMethodNames.ts
  • internal-packages/run-store/src/snapshotEntry.parity.test.ts
  • internal-packages/run-store/src/snapshotEntry.test.ts
  • internal-packages/run-store/src/snapshotEntry.ts
  • internal-packages/run-store/src/snapshotFaultInjection.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.test.ts
  • internal-packages/run-store/src/snapshotOrphanSweeper.ts
  • internal-packages/run-store/src/snapshotReadShapes.test.ts
  • internal-packages/run-store/src/snapshotReadShapes.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.births.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.off.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.readCohort.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.reads.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.staging.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.transitions.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointCycles.test.ts
  • internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointRecords.test.ts
  • internal-packages/run-store/src/testFixtures/snapshotIdFixture.ts
  • internal-packages/run-store/src/types.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 7 remain after this review.

📜 Review details
🔇 Additional comments (41)
internal-packages/run-store/src/snapshotOrphanSweeper.ts (8)

25-58: LGTM!


92-111: LGTM!


116-144: LGTM!


184-207: LGTM!


210-229: LGTM!


247-266: LGTM!


272-338: LGTM!


149-163: 🗄️ Data Integrity & Integration

No change needed. RunStore.findRunsByIds returns an ID-keyed Map, and its implementation keys entries by the internal id. rows.get(runId) uses the correct key.

internal-packages/run-engine/src/engine/tests/snapshotStoreChaos.test.ts (3)

32-58: LGTM!


60-104: LGTM!


226-279: LGTM!

internal-packages/run-store/src/delegatingRunStore.ts (1)

51-750: LGTM!

internal-packages/run-store/src/delegatingRunStore.test.ts (1)

76-88: 🎯 Functional Correctness

No change needed. runStoreMethodNames.ts checks both directions against keyof RunStore, so missing or extra member names cause a TypeScript error.

internal-packages/run-engine/src/engine/systems/executionSnapshotSystem.ts (1)

13-13: LGTM!

Also applies to: 452-452, 474-474, 497-497

internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.ts (1)

12-26: LGTM!

Also applies to: 49-124, 126-149

internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.test.ts (1)

9-28: LGTM!

Also applies to: 30-80, 82-178, 180-259, 264-332

internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointEquivalence.test.ts (1)

20-123: LGTM!

Also applies to: 125-330

internal-packages/run-engine/src/engine/tests/finalRunStatusParity.test.ts (1)

8-20: LGTM!

internal-packages/run-store/src/PostgresRunStore.snapshotId.test.ts (1)

10-149: LGTM!

internal-packages/run-store/src/PostgresRunStore.snapshotWrites.test.ts (1)

16-96: LGTM!

Also applies to: 98-202, 204-235, 237-311

internal-packages/run-store/src/snapshotOrphanSweeper.test.ts (1)

21-34: LGTM!

Also applies to: 37-117, 119-194, 196-272, 274-352, 354-396, 398-429

internal-packages/run-store/src/testFixtures/snapshotIdFixture.ts (1)

8-77: LGTM!

Also applies to: 79-149, 151-167

internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointRecords.test.ts (1)

25-109: LGTM!

Also applies to: 111-143, 147-161, 165-184, 186-218

internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.test.ts (1)

1839-1975: LGTM!

internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.ts (1)

509-563: LGTM!

Also applies to: 596-629

internal-packages/run-store/src/PostgresRunStore.snapshotTimestamps.test.ts (1)

113-132: LGTM!

internal-packages/run-store/src/taskRunExecutionSnapshotStore.reads.test.ts (1)

79-420: LGTM!

internal-packages/run-store/src/taskRunExecutionSnapshotStore.waitpointCycles.test.ts (1)

96-544: LGTM!

internal-packages/run-engine/src/engine/tests/helpers/decoratedStore.ts (1)

31-75: LGTM!

internal-packages/run-engine/src/engine/waitpointCoordinator/legacyPostgresCoordinator.ts (1)

104-149: 🗄️ Data Integrity & Integration

Do not add the COMPLETED predicate.

WaitpointSystem calls this method only after readRunBlockState reports all blockers as COMPLETED. Both reads pass the writer client, which routes to the owning primary, so a pending row cannot reach this mapping through the caller path.

internal-packages/run-store/src/redisSnapshotStore.ts (1)

32-44: LGTM!

Also applies to: 286-319, 426-426, 581-601, 617-623, 643-645, 680-680, 750-750, 762-762, 775-837, 867-900, 923-923, 949-957

internal-packages/run-store/src/taskRunExecutionSnapshotStore.ts (1)

118-138: LGTM!

Also applies to: 157-221, 227-257, 263-425, 447-566, 584-627, 636-651, 666-795, 804-866, 872-911, 913-962

internal-packages/run-store/src/snapshotFaultInjection.ts (1)

9-45: LGTM!

internal-packages/run-store/src/index.ts (1)

6-10: LGTM!

internal-packages/run-store/src/redisSnapshotStore.sinceCreatedAt.test.ts (1)

11-41: LGTM!

Also applies to: 43-193

internal-packages/run-store/src/snapshotReadShapes.ts (1)

13-27: LGTM!

Also applies to: 29-51, 53-95

internal-packages/run-store/src/taskRunExecutionSnapshotStore.off.test.ts (1)

10-38: LGTM!

Also applies to: 40-100

internal-packages/run-store/src/taskRunExecutionSnapshotStore.staging.test.ts (2)

20-65: LGTM!

Also applies to: 67-251


253-285: 📐 Maintainability & Code Quality

No change needed. PostgresRunStore.forWaitpointCompletion ignores the waitpoint ID and context and returns this, so an unknown waitpoint ID does not fail before the assertions.

internal-packages/run-store/src/snapshotReadShapes.test.ts (1)

7-17: LGTM!

Also applies to: 19-151

internal-packages/run-store/src/taskRunExecutionSnapshotStore.transitions.test.ts (1)

26-134: LGTM!

Also applies to: 136-454

🛑 Comments failed to post (8)
internal-packages/run-engine/src/engine/systems/waitpointSystem.ts (3)

490-497: 📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift

Add OTEL instrumentation for the completed-waitpoint propagation flow.

The new RunEngine flow reads completion envelopes and creates a snapshot without tracer or meter instrumentation. Add a trace span and bounded outcome metrics. Use only bounded attributes such as coordinator arm, snapshot status, and success or failure. Do not add run IDs, waitpoint IDs, or record counts as metric attributes.

  • internal-packages/run-engine/src/engine/systems/waitpointSystem.ts#L490-L497: instrument envelope collection and record-build outcomes.
  • internal-packages/run-engine/src/engine/systems/enqueueSystem.ts#L96-L117: instrument the completed-waitpoint snapshot handoff.

As per coding guidelines, “Integrate OpenTelemetry tracer and meter instrumentation in RunEngine systems for observability.”

📍 Affects 2 files
  • internal-packages/run-engine/src/engine/systems/waitpointSystem.ts#L490-L497 (this comment)
  • internal-packages/run-engine/src/engine/systems/enqueueSystem.ts#L96-L117

Source: Coding guidelines


744-773: 📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift

Add approved crumbs to the new snapshot propagation flow.

The new completion-envelope, record-building, and Redis-entry paths contain no crumb markers. Add // @Crumbs markers or `#region `@crumbs blocks with an approved namespace. Request a namespace before adding one because the provided guideline table is unavailable.

  • internal-packages/run-engine/src/engine/systems/waitpointSystem.ts#L744-L773: mark envelope classification and coordinator reads.
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.ts#L12-L60: mark record and output-variant selection.
  • internal-packages/run-store/src/snapshotEntry.ts#L135-L159: mark execution-snapshot entry construction.

As per coding guidelines, “Add crumbs as you write code” and “Do not invent new namespaces.”

📍 Affects 3 files
  • internal-packages/run-engine/src/engine/systems/waitpointSystem.ts#L744-L773 (this comment)
  • internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.ts#L12-L60
  • internal-packages/run-store/src/snapshotEntry.ts#L135-L159

Source: Coding guidelines


767-772: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

🔎 Supported by static analysis

🏁 Script executed:

printf '%s\n' '--- applicable repository conventions ---'
find /tmp/coderabbit-repo-knowledge/triggerdotdev-trigger-dev-0bdd0019 -type f -name '*.md' -print
printf '%s\n' '--- waitpointSystem structure and target ---'
ast-grep outline internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
sed -n '1,90p;700,785p' internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
printf '%s\n' '--- coordinator definitions and relevant methods ---'
rg -n --glob '*.{ts,tsx}' 'class (LegacyPostgresWaitpointCoordinator|StoreCoordinator)|readCompletionEnvelopes|LegacyPostgresWaitpointCoordinator' internal-packages

Repository: triggerdotdev/trigger.dev

Length of output: 15028


🏁 Script executed:

printf '%s\n' '--- applicable conventions and learnings ---'
cat /tmp/coderabbit-repo-knowledge/triggerdotdev-trigger-dev-0bdd0019/conventions/internal-packages-run-engine-src-engine-systems.md
cat /tmp/coderabbit-repo-knowledge/triggerdotdev-trigger-dev-0bdd0019/learnings/internal-packages-run-engine-src-engine-systems.md
printf '%s\n' '--- coordinator contract and implementations ---'
cat -n internal-packages/run-engine/src/engine/waitpointCoordinator/types.ts
sed -n '1,145p;490,585p' internal-packages/run-engine/src/engine/waitpointCoordinator/legacyPostgresCoordinator.ts
sed -n '1,90p;495,575p' internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.ts
printf '%s\n' '--- callers of the private read path and waitpoint ID parsing ---'
rg -n -A18 -B18 '`#completedWaitpointRecordsFor`|completedWaitpointRecordsFor|parseWaitpointId|b32hexW' internal-packages/run-engine/src/engine/systems/waitpointSystem.ts internal-packages/run-engine/src/engine/waitpointCoordinator internal-packages packages

Repository: triggerdotdev/trigger.dev

Length of output: 50381


🏁 Script executed:

printf '%s\n' '--- completion record builder and resume consumption ---'
cat -n internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointRecords.ts
sed -n '450,620p' internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
printf '%s\n' '--- StoreWaitpointCoordinator construction and usage ---'
rg -n -A12 -B8 'new Store|StoreWaitpoint|StoreCoordinator|storeCoordinator|WaitpointCoordinator' internal-packages/run-engine/src/engine --glob '*.ts'
printf '%s\n' '--- exact legacy envelope body ---'
sed -n '90,165p' internal-packages/run-engine/src/engine/waitpointCoordinator/legacyPostgresCoordinator.ts

Repository: triggerdotdev/trigger.dev

Length of output: 38971


🏁 Script executed:

printf '%s\n' '--- all uses of completedWaitpointRecords in the resume path ---'
rg -n -A12 -B12 'completedWaitpointRecords' internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
printf '%s\n' '--- store record read contract and coordinator capabilities ---'
rg -n -A20 -B10 'readCompletionEnvelopes|readRunBlockState|complete\\(|implements WaitpointCoordinator|export class WaitpointStoreCoordinator' internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.ts internal-packages/run-engine/src/engine/waitpointCoordinator/*.test.ts
printf '%s\n' '--- resources relevant to Redis/store routing ---'
rg -n -A8 -B8 'redisOptions|waitpoint.*Store|WaitpointStoreCoordinator|SystemResources' internal-packages/run-engine/src/engine/systems internal-packages/run-engine/src/engine --glob '*.ts'

Repository: triggerdotdev/trigger.dev

Length of output: 50382


🏁 Script executed:

sed -n '560,710p' internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
printf '%s\n' '--- targeted record propagation ---'
rg -n -A10 -B10 'completedWaitpointRecords' internal-packages/run-engine/src/engine/systems/waitpointSystem.ts
printf '%s\n' '--- store coordinator interface declaration ---'
sed -n '145,205p' internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.ts
rg -n 'readCompletionEnvelopes' internal-packages/run-engine/src/engine/waitpointCoordinator/storeCoordinator.test.ts

Repository: triggerdotdev/trigger.dev

Length of output: 12481


🏁 Script executed:

printf '%s\n' '--- completed waitpoint record contract and resolver ---'
rg -l 'CompletedWaitpointRecord|completedWaitpointRecords' internal-packages packages --glob '*.{ts,tsx}' | head -80
printf '%s\n' '--- relevant declarations and consumers ---'
rg -n -A18 -B12 'completedWaitpointRecords|CompletedWaitpointRecord' internal-packages/run-store packages internal-packages/run-engine/src --glob '*.{ts,tsx}' --glob '!**/*.test.ts' --glob '!**/*.bench.test.ts' | head -500

Repository: triggerdotdev/trigger.dev

Length of output: 50381


🏁 Script executed:

sed -n '1,130p' internal-packages/run-engine/src/engine/waitpointCoordinator/completedWaitpointResolver.ts
sed -n '584,670p' internal-packages/run-store/src/taskRunExecutionSnapshotStore.ts
rg -n -A18 -B12 'resolveCompletedWaitpoints|CompletedWaitpointResolver|completedWaitpointRecords' internal-packages/run-store/src internal-packages/run-engine/src --glob '!**/*.test.ts' --glob '!**/*.bench.test.ts'

Repository: triggerdotdev/trigger.dev

Length of output: 50381


Read store-format envelopes with WaitpointStoreCoordinator.

When a completed waitpoint has format b32hexW, waitpointSystem.ts:767 uses the fixed LegacyPostgresWaitpointCoordinator. A Redis-only waitpoint is omitted, and the resolver can throw UnresolvableWaitpointId with reason no-source. Keep legacy operations on Postgres, but route this envelope read to WaitpointStoreCoordinator. Add an integration test for a Redis-only b32hexW waitpoint.

internal-packages/run-engine/src/engine/tests/snapshotStoreReadGate.test.ts (1)

273-294: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

#!/bin/bash
# Description: Resolve the effective tsconfig strictness for the run-engine package.
set -euo pipefail

fd -t f 'tsconfig*.json' internal-packages/run-engine --exec cat {} \;
fd -t f 'tsconfig*.json' --max-depth 1 --exec cat {} \;

Repository: triggerdotdev/trigger.dev

Length of output: 391


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- target test excerpt ---'
sed -n '250,305p' internal-packages/run-engine/src/engine/tests/snapshotStoreReadGate.test.ts

printf '%s\n' '--- package tsconfig files ---'
fd -t f 'tsconfig*.json' internal-packages/run-engine -x sh -c 'echo "--- $1"; cat "$1"' sh {}

Repository: triggerdotdev/trigger.dev

Length of output: 2395


🏁 Script executed:

#!/bin/bash
set -euo pipefail

for file in \
  internal-packages/run-engine/tsconfig.src.json \
  internal-packages/run-engine/tsconfig.test.json \
  tsconfig.json
do
  if [ -f "$file" ]; then
    echo "--- $file"
    cat "$file"
  else
    echo "--- $file (missing)"
  fi
done

echo '--- tracked TypeScript configuration paths near run-engine ---'
git ls-files '*tsconfig*.json' | grep -E '(^|/)run-engine/|(^|/)tsconfig\.json$' || true

Repository: triggerdotdev/trigger.dev

Length of output: 3848


🌐 Web query:

TypeScript TS2454 variable used before being assigned assignment inside try finally definite assignment

💡 Result:

The TypeScript error TS2454, "Variable '...' is used before being assigned," occurs within try-finally blocks because TypeScript’s control flow analysis (CFA) must account for the possibility that code inside the try block could throw an exception at any point [1][2][3]. Consequently, the compiler cannot guarantee that an assignment statement inside a try block successfully executed before reaching the finally block or subsequent code [1][4][5]. ### Why This Happens When you declare a variable (e.g., let x: string;) without an initial value, TypeScript treats it as unassigned [3][5]. Even if you assign a value inside a try block, the compiler assumes that an error could occur before that assignment is reached [1][6]. Because the finally block or subsequent code might execute regardless of whether the try block completed, TypeScript flags the potential usage of an uninitialized variable as an error to prevent runtime undefined issues [3][5]. ### Recommended Solutions 1. Initialize at Declaration (Preferred) The most reliable way to resolve this is to provide an initial value at the point of declaration [3][7]. typescript // Fix: Provide a default value let x: string = ''; try { x = fetchData(); } finally { console.log(x); // Now guaranteed to be assigned } 2. Use a Union Type with undefined If a value is not immediately available, you can explicitly define the variable as potentially undefined [1][6]. This informs the compiler that the variable might not have a value, forcing you to handle the undefined case safely [1][6]. typescript let x: string | undefined; try { x = fetchData(); } finally { if (x !== undefined) { console.log(x); } } 3. Definite Assignment Assertion If you are certain that the variable will be assigned before it is used—and you want to bypass the compiler check—you can use the definite assignment assertion operator (!) [3][5]. Use this sparingly, as it tells the compiler to trust you, potentially hiding actual bugs [7]. typescript let x!: string; try { x = fetchData(); } finally { console.log(x); // Compiler accepts this } In summary, TS2454 is a safeguard against unpredictable control flow [3]. By initializing variables or explicitly handling undefined states, you align your code with TypeScript's safety requirements [3][7].

Citations:


Initialize runId before the try block.

The test project enables "strict": true. TypeScript can report TS2454 because runId is assigned inside try and read after finally. Initialize it or restructure the code so the assignment is definite.

internal-packages/run-store/src/delegatingRunStore.forwarding.test.ts (1)

11-17: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Add a crumb marker for this new source-level test.

Add // @Crumbs or a `#region `@crumbs block near the source parsing setup. This file currently has no required crumb instrumentation.

As per coding guidelines, “Add crumbs as you write code — not just when debugging.”

Source: Coding guidelines

internal-packages/run-store/src/PostgresRunStore.ts (1)

2033-2059: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

#!/bin/bash
# Description: List TaskRunExecutionSnapshot scalar columns and compare with the echoed literal.
set -euo pipefail

fd -t f 'schema.prisma' --exec rg -n -A 60 'model TaskRunExecutionSnapshot\b' {} \;

fd -t f 'snapshotEntry.ts' internal-packages/run-store/src --exec cat -n

Repository: triggerdotdev/trigger.dev

Length of output: 11760


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- PostgresRunStore context ---'
sed -n '1970,2085p' internal-packages/run-store/src/PostgresRunStore.ts

printf '%s\n' '--- SnapshotEntryInput and related hydrator definitions ---'
rg -n -C 8 'type SnapshotEntryInput|interface SnapshotEntryInput|SnapshotEntryInput|lastHeartbeatAt|entryFromCreateExecutionSnapshot' internal-packages/run-store/src

Repository: triggerdotdev/trigger.dev

Length of output: 50381


Add lastHeartbeatAt: null to the Redis-only snapshot echo.

TaskRunExecutionSnapshot declares this nullable column, and the Redis hydrator returns null for it. The echo omits it and bypasses type checking, so callers receive undefined instead of null.

internal-packages/run-store/src/taskRunExecutionSnapshotStore.readCohort.test.ts (1)

20-24: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

Remove the mocked RunStore.

The empty-object cast is a mocked RunStore. Extract the cohort predicate into a pure function and test that function, or construct the store with a testcontainer-backed RunStore.

As per coding guidelines, “We use vitest exclusively. Never mock anything - use testcontainers instead.”

Source: Coding guidelines

internal-packages/run-store/src/taskRunExecutionSnapshotStore.ts (1)

746-754: 🚀 Performance & Scalability | 🟡 Minor | ⚡ Quick win

One extra Redis round trip per non-head row in the window read.

getSinceCreatedAt attaches completedWaitpointIds to the head row only. Every other row therefore falls into the await this.redis.getSnapshotWaitpointIds(runId, read.id) branch in #hydrate, so a window of take rows costs up to take - 1 extra Redis calls where Postgres served the same window with one query. The comment on Line 749 states that each row carries its own order, which the store does not currently provide.

Two options: return per-row order from the Lua script, or accept an empty order for non-head rows and say so in the comment. If the engine only reads completedWaitpointOrder from the head row, the second option removes the fan-out entirely.

Also applies to: 820-826

devin-ai-integration[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

d-cs added 9 commits September 2, 2026 10:22
…h coordinator arms

The resume path only has id, status, type and completedAfter per edge, which is nine
fields short of a completion envelope. Add one coordinator method that sources the rest,
implemented by both arms so the record build never branches on residency.

The store arm reads wp:{id} alone: both halves live under that key, so one pipelined
HMGET per id needs no run-scoped key and cannot span two cluster slots. An id with no
record, or a record with no completion, is omitted rather than defaulted.
One record per distinct id. The ordered id list carries multiplicity and holds only
batch-indexed ids, so the record set is what says which waitpoints completed.

The output variant is chosen, never copied: an offloaded value stays a reference, a plain
RUN output becomes a marker re-read from TaskRun.output, a BATCH output is omitted because
the runtime discards it at source, and everything else rides inline under the pre-existing
thresholds. No new cap and no completion-time spill.

A RUN error and an orphaned RUN both stay inline. TaskRun.error is jsonb and does not
round-trip, and the completing-run back-reference nulls on delete.
Rebuilds CompletedWaitpoint[] from a wait cycle's ordered id list and records, field-for-
field equivalent to the existing snapshot hydration, which is what the executor consumes.

It iterates the records, never the order. The order holds only batch-indexed ids, so
iterating it would drop every index-less wait: each wait.for, each single triggerAndWait
and each token. The equivalence suite pins that, and fails on 10 of 12 cases if the
iteration is inverted.

The coverage check is the fail-loud rule. The id classifier is total and never throws, so
an unrecognised shape would otherwise classify as legacy, find no row, and vanish from the
resumed run's completed set. An id that no half resolves throws, and so does an id that
both halves claim.
…at the resume appends

Carries an envelope per distinct id from the resume path into the wait cycle's key, filling
the hole the snapshot store left for this lane. The records ride the mint only: a
copy-forward writes no key and needs none.

continueRunIfUnblocked builds the set once and passes it at both appends. The build is
gated on id shape, so a wait with no store-resident half supplies no records and a
Postgres-resident resume is byte-identical to before. Nothing mints a store-format
waitpoint yet, so every live path supplies none today.

The existing waitpoint corpus passes unmodified.
The base branch gained a refusal path: when the store declines an untrustworthy cycle
pointer it mints a replacement inside the same call, from the refs the caller carried. That
replacement needs the records too. A cycle holding ids with no records makes the resolver's
coverage check reject a legitimate resume, because every distinct id must resolve through
exactly one half.

Also pins the no-refs case, where writing no pointer at all stays correct.
… envelope

Coverage check now runs over the whole membership, not the order. The order omits every
index-less wait by construction, so an order-scoped check could not see an index-less id
whose record was missing — the exact loss the resolver exists to prevent. Adds distinctIds
to the resolver args and updates the jointly-owned freeze pin.

A refused copy-forward no longer mints a records-less cycle. Copy-forward appends carry no
records of their own, and the append script can refuse a pointer and mint a replacement from
the carried refs, so the decorator reads the surviving cycle's records and carries those.

A deriveFromRun record whose run output is gone now fails loud instead of resolving to an
empty output. Postgres does not lose it: the back-reference nulls on delete but the stored
output stays, so returning undefined would resolve a triggerAndWait with silently wrong data.

The legacy arm passes the routing hint it was dropping, so a resume reads the run's own store
instead of fanning out across every run-ops database, and reuses the chunked fetch rather than
reading a large fan-in whole.

The envelope read issues one command per id concurrently rather than as a pipeline. Each id is
its own hash tag, so N ids are N cluster slots and a pipeline spanning them is rejected under
cluster mode — which a single-node test server would never surface.

Also: shares one row-to-source mapper between the legacy arm and the equivalence suite, so a
bug in the arm can no longer hide from the oracle; pins the deliberate BATCH-output divergence
and corrects the comment that gave the wrong reason for it; gates the record build on id
format rather than claiming residency; and builds the record set inside the two branches that
append rather than before the statuses that return without appending.
The store arm cannot return a pending waitpoint, because a pending one has no completion to
read. The legacy arm read rows by id with no status filter, so it could hand back an envelope
for a PENDING waitpoint with completedAt defaulted to now. The resolver's coverage check reads
an omission as "fail loud", so the arms disagreeing there would turn a pending waitpoint into
a resumable one. Filters to COMPLETED.

Also states why the ref branch precedes the RUN branch, which is the opposite order to the
reference implementation in the freeze test. Both are byte-identical at read time by that
reference's own reasoning, and this order needs no Postgres read to recover a string already
in hand — and keeps an offloaded RUN success resolvable when the completing run row is gone,
which now refuses rather than resolving empty. Adds the offloaded-RUN-success case that both
suites were missing.
Replaces the hand-written run-output callbacks with a real Postgres read. The branch's premise
is that TaskRun.output holds the same string the waitpoint carried, and only a real row can
settle that — a callback returning a literal asserted that the callback was called.

Adds createRunOutputReader, the production reader over the store, so the read routes to the
run's owning database. The dependency is now optional, because most cycles carry no
deriveFromRun record; one that does with no reader wired throws, since that is a wiring error
rather than a data condition.

The equivalence suite runs against seeded child runs whose output matches each RUN row, so the
parity claim is now checked end to end rather than against a value the test supplied twice.
The pure suite keeps every case that performs no read and is built with no reader at all.

One wrapper remains, and delegates to the real reader: it counts reads to pin one query per
record rather than one per batch index, which the resolved output cannot show.
@d-cs
d-cs force-pushed the feat/waitpoint-envelope-resolver-tri-13441 branch from 5b4b060 to 9e8dc9d Compare September 2, 2026 09:24
devin-ai-integration[bot]

This comment was marked as resolved.

d-cs and others added 4 commits September 2, 2026 18:14
The carry-forward records read fired on every copy-forward append that carried waitpoints,
because no caller supplies records yet. That is a Redis round trip on the resume path for
every organisation whose snapshot-store dial is on, including those holding no store-format
waitpoint at all.

A record set is written for store-format waitpoint ids and nothing else, so a cycle whose ids
are all legacy has none and the read could only return nothing. Gating on the id keeps a
deployment with no store-format waitpoint at zero extra round trips, and confines the cost to
the organisations that actually hold them.

This is the one place the snapshot store looks inside a waitpoint id rather than treating the
record set as opaque. Two tests count the reads rather than infer them, since the cost is the
point of the gate and is invisible in the resulting entry.
The gate was an inline predicate inside a private method, so the cost decision it drives had
only integration coverage. Extracts it as mayHoldRecords beside deriveOrder and
deriveDistinctIds, its sibling pure helpers, and covers it with unit tests: empty, all-legacy,
one store id, a store id among legacy ones, index-less, all four waitpoint types, and a
foreign prefix.

The mixed case is the one a per-organisation rollout produces, where a run holds waitpoints
minted either side of the flip. One store-format id is enough to require the read.
The store arm launched one HMGET per waitpoint through a single Promise.all, so a 1000-item
batch fan-in would issue 1000 concurrent commands at once. Each is small, but the burst is
the cost.

Chunks at 100, matching the bound the snapshot hydration already uses for the same shape of
read. Stays one command per key, so nothing can span two cluster slots and the single-slot
assertion is still unnecessary here.

Two tests cover the chunk boundary at 250 and a mixed complete/pending set at 150, since an
increment that disagrees with the slice is the realistic failure of hand-rolled chunking and
is invisible in a single-chunk test.
…d path

The append script now sources a refused carry-forward's record set from the
cycle it replaces, rather than the caller pre-reading it and passing it in.
The pre-read cost one HGET plus a parse of the whole record blob on every
copy-forward append -- attempt start, dequeue, checkpoint -- to serve a branch
that needs partial eviction to reach. Reading it in the branch that uses it is
also atomic with the mint, so no reader can observe a replacement cycle whose
ids have no records.

Projects the legacy arm's envelope read to the columns the envelope is built
from, so it stops reading tags and the ownership and timestamp columns it does
not use on a read that lands on the writer inside the run lock.

Corrects a docstring that described the extra read as happening on the resume
path: the resume path supplies the records and was the one path that never
read. Records the rollout-ordering constraint that keeps the format gate cheap.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

d-cs and others added 2 commits September 3, 2026 11:15
…positions

Two costs in the completed-waitpoint resolver, both scaling with fan-in width,
which is the input the record set exists to make cheap.

The output hydration read one run per record, awaited in series inside the emit
loop. A batch parent resuming on 500 children therefore performed 500
sequential Postgres reads, where the hydration it replaces does one chunked
findMany. The reader is now plural: the resolver collects the distinct set of
runs its records defer to, reads them in one chunked batch before the loop, and
hydrates from the resulting map. A cycle that defers nothing reads nothing.

Positions came from a linear scan of the order per record, making the emit loop
quadratic -- about a million string comparisons for a thousand-wide wait. The
order is now indexed once into a map of id to positions. Absence still carries
the index-less case, so a wait with no batch index emits one entry with an
undefined index exactly as before.

Neither path has a production caller yet, so nothing changes for any
organisation today.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ver's absence rules

Two reviews of the resolver rewrite raised four things worth acting on.

The resume-path gate built a mapped array, a filtered array and a Set before
discovering there was nothing to ask for, which is the universal case today and
stays the common one through a partial rollout. It now short-circuits on .some
first, so the gate allocates nothing until a blocking waitpoint really is
store-format, and stops scanning at the first one that is.

The run-output reader no longer accepts a read client. The router reads the
owning store's replica when none is passed and forces its primary for any
client that is not replica-branded, so accepting one invited a wide output read
onto the writer by reflex. This read cannot need read-your-writes: the child
committed its output before completing the waitpoint that unblocked the parent.

Three absence rules had no test holding them. An empty-string output is a
value, and the reader's explicit null check is what keeps it from being
reported as a lost output on a run that completed normally. Two waitpoints
completed by the same run must read it once and both still receive it. And the
differential suite never compared a cycle with two deferring records, so the
batched read had no oracle covering the shape it exists for.

Also throws the missing-reader error where the deferring record is found, so
the message names a record that really defers and needs no optional chaining.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

d-cs and others added 6 commits September 3, 2026 11:55
A copy-forward append learns from the head probe whether its waitpoint id set
continues the previous cycle. When that probe threw, the append fell back to
minting a fresh cycle -- correct, because an unverified pointer must not be
carried -- but a copy-forward holds no records of its own, and the probe was
what would have found the previous cycle to read them from. So the mint wrote
waitpoint ids with no records behind them, and the next resume refused the
whole cycle rather than lose a result silently. One transient probe failure
left a run unable to resume, with its rows still in Postgres.

Reachable with the store healthy, because the probe parses the entry payload:
a single corrupt entry is enough.

The mint now inherits the previous cycle's records when the distinct id set is
identical, which is the comparison the probe would have made had it succeeded.
It happens inside the append script, so the read is atomic with the mint and
costs nothing on any path that does not fail. A differing id set is a genuinely
new wait and still starts with the caller's own records, even when that is
none, so a run cannot be handed a result for a waitpoint it is not waiting on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… a string

The inherit guard compared the two serialised id arrays. Both are derived from
Postgres reads with no ORDER BY -- the resume reads the run's block edges, the
copy-forward reads the snapshot's waitpoint rows -- so the same set of ids
arrives in an arbitrary order on each append. String equality therefore failed
for almost every wait holding two or more waitpoints, and the mint fell back to
writing a cycle with no records: the exact state the inherit exists to prevent,
missed for the batch fan-in it matters most for.

It now decodes both sides and compares membership, which is what the
decorator's own sameSet does and for the same reason.

The test that was meant to hold this used a single waitpoint, the one case
where string and set comparison agree, so it passed for the wrong reason. It
now uses two waitpoints and re-passes them in the opposite order, and fails
against the string comparison.

Also corrects the guard's comment. It is deliberately weaker than the carry
test the decorator applies on a successful probe, which additionally requires
the order to match: a pointer hands the reader the previous cycle's order,
whereas this mints a fresh cycle from the caller's own, so only the records
have to be right and records are keyed by waitpoint id.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The envelope build and the resolver were covered separately and the join
between them was not. The run-store suite proves records reach the cycle key,
then reads them back with a raw probe. The equivalence suite proves the
resolver reproduces the existing hydration, from records built by hand.
Neither runs write, read and resolve in one pass, so an envelope emitted in a
shape the resolver does not expect passes both and fails in neither.

This sources envelopes from real waitpoint rows through the arm the engine
actually constructs, writes them through the real store, reads the id lists
back through the store's own read API, resolves, and compares against
enhanceExecutionSnapshotWithWaitpoints over the same rows.

Five cases: an inline output, a RUN output the record defers to its run rather
than carrying, a mixed legacy and store-format snapshot where each half reads
its index from the same order, one waitpoint at two batch indexes, and a member
id whose records were lost, which must refuse rather than resume short.

The records field is read with a probe because no read API for it exists yet;
that hydration belongs to the snapshot-store lane. Everything either side of
that read is production code, which the tests pin: emptying the arm's envelope
build fails four of the five.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… output on the writer

Two defects in the deferred-output path, both found in review.

A task that returns nothing completes its waitpoint with no output at all. The
record build still marked it derivable, so the resolver read a null
TaskRun.output and refused the resume as a lost output -- for every
triggerAndWait on a void task. The hydration this replaces resumes cleanly with
no output. A record now defers only when the waitpoint actually carried an
output; one that did, whose run row has since gone, still refuses, which is
what the refusal is for. An empty string stays derivable, because empty is a
value.

A test asserted the broken behaviour was correct, which is why nothing caught
it. It now pins the case it meant to cover: a record that defers to a run whose
output is gone.

The run-output read also needs read-your-writes, and an earlier revision
removed its client to keep the read off the writer. The ordering argument for
that was wrong: the child does commit its output before completing the
waitpoint, but the reader can observe the completion by another route while the
run-output replica still trails, and the output then reads null and the resume
is refused. A hard refusal is worse than a read on the writer, and this read is
bounded and projected to one column. The client is required rather than
optional, because the router turns a missing client into a replica read and an
optional parameter would reintroduce the window whenever a caller omitted it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ed-output fail-open

An exhaustive walk of the deferred-output state space found the mechanism
sound, and asked for one change plus two cheap ones.

The required-writer rule could not be enforced by the type system. ReadClient
admits both a writer and a replica, and the two are structurally identical --
separable only by a runtime brand -- so a caller passing readOnlyPrisma would
type-check and quietly reinstate the replica-lag window that turns a committed
child output into a refused resume. The reader now asserts the brand. It is
built once at wiring time, so the check costs nothing per read.

A record marked as deferring its output while carrying no run id was the one
place this design failed OPEN rather than loud: it resolved the waitpoint with
no output and no error. Unreachable from the current record build, which
requires the run id before it marks anything derivable, but a reordering of
those conditions is all it would take. It now throws, like a record arriving
with no reader wired.

Also pins the orphan case against the oracle: a RUN waitpoint whose completing
run is gone keeps its own output inline and must omit the run sub-object, which
only the differential comparison catches.

Records the premise the deferred read rests on: the completing run's output
still holds what the waitpoint was completed with. Nothing overwrites it today,
and if that changes the divergence would be silent rather than loud.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…an-in scan

Deciding whether to build a completed-waitpoint record set meant walking every
blocking waitpoint and classifying its id. Because no id is store-format today,
every resume for every organisation walked its whole blocker list to reach the
same answer: nothing to build. No allocation and no I/O, but not nothing, and
proportional to fan-in on exactly the wide waits this feature exists to serve.

An injected predicate now answers that in constant time, ahead of the scan. It
has to be injected rather than derived in the engine: the answer is
per-organisation, following the snapshot store's own rollout, and the store
keeps that state private. It defaults to never, so no resume does any of this
work until the ticket that wires the store supplies it.

The id-format scan stays behind the predicate, because it is what keeps a mixed
cycle working once records are enabled: an organisation mid-rollout holds
waitpoints minted either side of the flip, and the format is the only thing
that says which half an id belongs to.

Threaded through the engine options so it is settable and testable rather than
unreachable configuration.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

Devin Review

Comment on lines +903 to +907
local carried = records
if carried == '' then
carried = redis.call('HGET', wpKey(cycleSeqIn), 'records') or ''
end
cycleSeq = mintCycle(carried)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Evicted cycle blocks resumed runs

If a cycle disappears after a copy-forward probe, mintCycle creates its replacement without records. Later reads treat it as complete and reject every resume.

Prompt for agents
Handle the carry-forward refusal where wp:<cycleSeqIn> disappeared after the decorator's successful probe and the caller supplied no records. The current Lua branch mints a non-dangling replacement containing distinct IDs but no records, so the completed-waitpoint resolver's coverage check permanently rejects it instead of triggering the Postgres fallback. Change the append protocol so this race cannot publish an authoritative record-less cycle. Possible approaches include returning a retry/repair outcome without writing the new head, or explicitly marking the replacement as unresolved so reads fall back to Postgres. Update the refusal tests to cover a later resolver/read, not only the raw absence of records.
Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

d-cs and others added 2 commits September 3, 2026 15:25
… properly

The gate took only a run id. The decision it answers is organisation-scoped --
it follows the snapshot store's own rollout -- and an opaque run id cannot
answer that without a lookup, which would put a read back on the path the gate
exists to keep free. Both call sites already hold the organisation on the
snapshot they are transitioning from, so it now travels with the run id and
costs nothing.

The test claimed "once per resume" and did not show it. It blocked the run on a
single waitpoint, where a gate consulted once per waitpoint is
indistinguishable from one consulted once per resume, and asserted containment
rather than a count -- so it passed whether the gate was called once, three
times, or after the scan. It now blocks on three waitpoints and asserts exactly
one consultation carrying the run's own organisation. A per-waitpoint gate
fails it three-to-one, and passing the wrong id fails it outright.

What the test still does not assert is that the gate runs BEFORE the id scan.
That is not observable from outside: the scan is a pure function over ids the
caller already holds, so it leaves no trace, and proving the negative would
need it stubbed. The ordering is held by the code, where the predicate is the
method's first statement, and is recorded as such in the test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The gate took an options object, so every resume allocated a literal to ask a
question that is answered no for every organisation today. Two positional
arguments make the disabled path allocate nothing, which is what the earlier
claim of costing nothing needed in order to be true rather than nearly true.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant